Robot operation for a moving workpiece

ABSTRACT

A robot system includes: a conveying device configured to convey a workpiece; a robot configured to execute an operation on the workpiece; and circuitry configured to: identify a current position of the workpiece and an object area occupied by an object; identify an interlock area that moves with the current position of the workpiece being conveyed by the conveying device; check an overlap between the interlock area and the object area; and control the robot to execute the operation based on the current position of the workpiece in response to determining that the interlock area does not overlap the object area.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of PCT Application No.PCT/JP2020/047245, filed on Dec. 17, 2020, which claims priority fromU.S. Provisional Application No. 62/948,820, filed on Dec. 17, 2019, theentire contents of which are incorporated herein by reference.

BACKGROUND Field

The present disclosure relates to a robot system, a controller, and acontrol method.

Description of the Related Art

Japanese Unexamined Patent Publication No. 2007-148527 discloses a robotinterference avoidance method in which, for each of a plurality ofrobots, in a case where there is an interference area in which occupyingarea of the robot itself overlaps with an occupying area of anotherrobot, the robot stops operation before entering the interference areainside its own occupying area when detecting that the other robot entersthe interference area through communication with the other robot, andthe robot resumes operation of moving to its own target position whenthe other robot exits from its own occupying area.

SUMMARY

Disclosed herein is a robot system. The robot system may include: aconveying device configured to convey a workpiece; a robot configured toexecute an operation on the workpiece; and circuitry configured to:identify a current position of the workpiece and an object area occupiedby an object; identify an interlock area that moves with the currentposition of the workpiece being conveyed by the conveying device; checkan overlap between the interlock area and the object area; and controlthe robot to execute the operation based on the current position of theworkpiece in response to determining that the interlock area does notoverlap the object area.

Additionally, a controller is disclosed herein. The controller mayinclude a circuitry configured to: identify an interlock area that moveswith a current position of a workpiece being conveyed by a conveyingdevice; check an overlap between the interlock area and an object areaoccupied by an object; and control a robot to execute an operation onthe workpiece based on the current position of the workpiece in responseto determining that the interlock area does not overlap the object area.

Additionally, a control method is disclosed herein. The control methodmay include: identifying an interlock area that moves with a currentposition of a workpiece being conveyed by a conveying device; checkingan overlap between the interlock area and an object area occupied by anobject; and controlling a robot to execute an operation on the workpiecebased on the current position of the workpiece in response todetermining that the interlock area does not overlap the object area.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an example overallconfiguration of a robot system.

FIG. 2 is a schematic diagram illustrating an example configuration of arobot.

FIG. 3 is a block diagram illustrating an example configuration of anenvironment identification device and a robot controller.

FIG. 4 is a schematic diagram illustrating an example occupying area ofa robot.

FIG. 5 is a schematic diagram illustrating an example occupying area ofa worker.

FIG. 6 is a table illustrating example progress data.

FIGS. 7A and 7B are schematic diagrams illustrating an example interlockarea.

FIG. 8 is a schematic diagram illustrating an example operation of therobot system where the interlock area and the occupying area overlap.

FIG. 9 is a schematic diagram illustrating an example operation of therobot system where the interlock area and the occupying area overlap.

FIG. 10 is a schematic diagram illustrating an example operation of therobot system where the interlock area and the occupying area do notoverlap.

FIG. 11 is a schematic diagram illustrating an example operation of therobot system where the interlock area and the occupying area do notoverlap.

FIG. 12 is a block diagram illustrating an example hardwareconfiguration of the environment identification device and the robotcontroller.

FIG. 13 is a flowchart illustrating an example environmentidentification procedure.

FIG. 14 is a flowchart illustrating an example robot control procedure.

DETAILED DESCRIPTION

In the following description, with reference to the drawings, the samereference numbers are assigned to the same components or to similarcomponents having the same function, and overlapping description isomitted.

Robot System

A robot system 1 illustrated in FIG. 1 is a system for producing aworkpiece by cooperation of a conveying device and at least one robot.Hereinafter, in the production process of the workpiece, all objectsthat are operation targets of each local device are referred to as“workpieces”. For example, the term “workpiece” includes a final productin the robot system 1, a component of the final product, and a unitobtained by combining a plurality of components. As an example, therobot system 1 includes a conveying device 2, a plurality of (forexample, three) robots 3A, 3B, and 3C, and a control system 100.

The conveying device 2 conveys a workpiece 90 by the power of anelectric motor or the like, for example. As an example, the conveyingdevice 2 conveys the workpiece 90 in one direction along a horizontalplane. Specific examples of such the conveying device 2 include a beltconveyor and a roller conveyor.

Each of the robots 3A, 3B, and 3C performs a predetermined operation onthe workpiece 90 conveyed by the conveying device 2. The robots 3A, 3B,and 3C may include two or more robots whose operation ranges overlapeach other. As an example, the robot 3B performs operation on theworkpiece 90 in an operation range overlapping an operation range of therobot 3A, and the robot 3C performs operation on the workpiece 90 in anoperation range overlapping the operation range of the robot 3A and theoperation range of the robot 3B. Specific examples of the operationperformed on the workpiece 90 include assembly of another workpiece 90(for example, a sub-part) to the workpiece 90 (for example, a base part)conveyed by the conveying device 2, and fastening (for example, boltfastening) and joining (for example, welding) of parts in the workpiece90 conveyed by the conveying device 2.

At least one of the robots 3A, 3B, and 3C may be a robot capable ofautonomous traveling. FIG. 1 illustrates a case where all of the robots3A, 3B, and 3C are capable of autonomous traveling. As illustrated inFIG. 2, each of the robots 3A, 3B, and 3C includes a robot body 4 and aconveying carriage 5. The conveying carriage 5 (a carriage) holds therobot body 4 and autonomously travels around the conveying device 2 tocarry the robot body 4. Specific examples of the conveying carriage 5include so-called electric automated guided vehicles (AGVs).

The robot body 4 is a six-axis vertical articulated robot, and includesa base 11, a turning part 12, a first arm 13, a second arm 14, a thirdarm 17, a tip 18, and actuators 41, 42, 43, 44, 45, and 46.

The base 11 is fixed on the conveying carriage 5. The turning part 12 isprovided on the base 11 so as to pivot about a vertical axis 21. Thefirst arm 13 is connected to the turning part 12 so as to swing about anaxis 22 that intersects (for example, is orthogonal to) the axis 21.Intersecting includes a case where there is a twisted relationship suchas so-called three-dimensional crossing. The second arm 14 is connectedto the tip of the first arm 13 so as to swing about an axis 23 that issubstantially parallel to the axis 22. The second arm 14 includes an armbase 15 and an arm end 16. The arm base 15 is connected to the tip ofthe first arm 13 and extends along an axis 24 that intersects (forexample, is orthogonal to) the axis 23. The arm end 16 is connected tothe tip of the arm base 15 so as to pivot about the axis 24. The thirdarm 17 is connected to the tip of the arm end 16 so as to swing about anaxis 25 that intersects (for example, is orthogonal to) the axis 24. Thetip 18 is connected to the tip of the third arm 17 so as to pivot aboutan axis 26 that intersects (for example, is orthogonal to) the axis 25.

As described above, the robot body 4 includes a joint 31 connecting thebase 11 and the turning part 12, a joint 32 connecting the turning part12 and the first arm 13, a joint 33 connecting the first arm 13 and thesecond arm 14, a joint 34 connecting the arm base 15 and the arm end 16in the second arm 14, a joint 35 connecting the arm end 16 and the thirdarm 17, and a joint 36 connecting the third arm 17 and the tip 18.

The actuators 41, 42, 43, 44, 45, and 46 include, for example, anelectric motor and a speed reducer, and drives the joints 31, 32, 33,34, 35, and 36, respectively. For example, the actuator 41 turns theturning part 12 around the axis 21, the actuator 42 swings the first arm13 around the axis 22, the actuator 43 swings the second arm 14 aroundthe axis 23, the actuator 44 turns the arm end 16 around the axis 24,the actuator 45 swings the third arm 17 around the axis 25, and theactuator 46 turns the tip 18 around the axis 26.

The tip 18 is provided with a work tool. Specific examples of the worktool include a holding tool that holds an object by gripping or suction,a screw tightening tool, and a welding torch. The robot body 4 adjuststhe position and the posture of the work tool within a movable range bydriving the joints 31, 32, 33, 34, 35, and 36 with the actuators 41, 42,43, 44, 45, and 46, respectively.

Specific configurations of the robots 3A, the 3B, and the 3C may bemodified. For example, each of the robots 3A, 3B, and 3C may be aseven-axis redundant robot in which one joint is added to the six-axisvertical articulated robot, or may be a so-called SCARA type articulatedrobot. The robots 3A, 3B, and 3C may not be capable of autonomoustraveling, and the base 11 may be fixed around the conveying device 2.

Each of the robots 3A, 3B, and 3C may further include an operationtarget camera 51. The operation target camera 51 (an operation targetimaging device) images an operation area of the robot body 4 to generateoperation target image data. Specific examples of the operation targetcamera 51 include an electronic camera including a solid-state imagingdevice such as a charge coupled device (CCD) image sensor or acomplementary metal-oxide-semiconductor (CMOS) image sensor and anoptical system that forms an image of the operation area on the imagingdevice.

For example, the operation target camera 51 is attached to the tip 18together with the work tool. Accordingly, when the work tool is directedto the operation area, the operation target camera 51 can be directed tothe operation area, and the operation area can be more reliably imaged.The installation position of the operation target camera 51 may not belimited to the tip 18. The operation target camera 51 may be provided inany part of the robots 3A, 3B, and 3C as long as the operation area canbe imaged.

The robot system 1 may further comprise an environment imaging device 6.The environment imaging device 6 images a target area including theworkpiece 90 conveyed by the conveying device 2 to generate environmentimage data. The environment image data may be of an imaging targetincluding the workpiece conveyed by the conveying device 2 and theobject. For example, the environment imaging device 6 images theconveying device 2 from vertically above.

The environment imaging device 6 may be configured to generate aplurality of types of image data. For example, the environment imagingdevice 6 may generate environment image data including distance imagedata for the target area and luminance image data. The luminance imageis a two-dimensional image in which the brightness of each part in thetarget area is represented by a color distribution. The distance imageis a two-dimensional image in which the distance to each part in thetarget area is represented by a color distribution. Representing by thecolor distribution means representing by a distribution of at least oneof three attributes of color (hue, brightness, and saturation).

For example, the environment imaging device 6 includes a luminancecamera 7 that generates the luminance image data and a distance camera 8that generates the distance image data. Specific examples of theluminance camera 7 include an electronic camera including a solid-stateimaging device such as a charge coupled device (CCD) image sensor or acomplementary metal-oxide-semiconductor (CMOS) image sensor and anoptical system that forms an image of the operation area on the imagingelement. Specific examples of the distance camera 8 include atime-of-flight (TOF) camera that obtains range information based on thetime from when infrared light or the like is emitted to the target areauntil the reflection light returns. For example, the distance image datamay include first pixels each of which is associated with a distancefrom the distance camera 8 to a portion of the imaging target. theluminance image data may include second pixels each of which isassociated with a luminance of a portion of the imaging target. Theenvironment imaging device 6 may not generate a plurality of types ofimage data, and may be configured to generate luminance image data.

The control system 100 (circuitry) include an environment identificationdevice 200, a robot controller 300, and a conveyance controller 400. Theconveyance controller 400 controls the conveying device 2 so as toconvey the workpiece 90 at a predetermined speed, for example. Theenvironment identification device 200 identifies the current position ofthe workpiece 90 in the conveying device 2 and one or more occupyingareas of one or more objects in the vicinity of the workpiece 90. Forexample, the one or more occupying areas include an object area occupiedby an object. The one or more objects may include the robots 3A, 3B, and3C. The one or more objects may include a device other than the robots3A, 3B, and 3C, or may be a human worker (person) who cooperates withthe robots 3A, 3B, and 3C.

The environment identification device 200 may be configured to identifya current position of the workpiece 90 and an occupying area of the oneor more objects based on the image data generated by the environmentimaging device 6. For example, as illustrated in FIG. 3, the environmentidentification device 200 includes an image data acquisition unit 211,an environment image data storage unit 212, a workpiece positiondetection unit 213, and an occupying area detection unit 214 asfunctional components (hereinafter referred to as “functional block”).

The image data acquisition unit 211 acquires the environment image datagenerated by the environment imaging device 6. The environment imagedata storage unit 212 stores the environment image data acquired by theimage data acquisition unit 211. The workpiece position detection unit213 identifies the current position of the workpiece 90 based onenvironment image data. The workpiece position detection unit 213 mayidentify the current position of the workpiece 90 based on the luminanceimage data. The workpiece position detection unit 213 recognizes themarker attached to the workpiece 90 based on the luminance image data,and identifies the current position of the workpiece 90 based on theposition of the marker in the target area (the imaging target area bythe environment imaging device 6).

FIG. 4 is a view of the conveying device 2 viewed from vertically above,and the environment imaging device 6 images a target area TA1 in thefigure. The workpiece 90 includes a plurality of operation target parts92. The operation target parts 92 are a target site of operationperformed by the robots 3A, 3B, 3C, or the worker. There is noparticular limitation on the operation performed on the operation targetpart 92. For example, the robots 3A, 3B, 3C, or the worker performs anoperation of assembling a component to the operation target part 92.

A marker 91 is attached to the workpiece 90. For example, the marker 91is attached to a position that is not included in the plurality of theoperation target parts 92 in the workpiece 90. The marker 91 may beanything as long as it is recognizable by image processing. Specificexamples of the marker 91 include a two-dimensional bar code. In theexample of FIG. 4, the workpiece position detection unit 213 identifiesthe current position of the workpiece 90 based on the position of themarker 91 in the target area TA1. The workpiece position detection unit213 may identify the current position of the workpiece 90 by recognizinga feature on the workpiece 90 (for example, circular unevenness) or theworkpiece 90 itself through image processing instead of the marker 91.

Returning to FIG. 3, the occupying area detection unit 214 identifiesthe occupying area of one or more objects based on the environment imagedata. The occupying area detection unit 214 may identify the currentposition of the workpiece 90 based on the distance image data and theluminance image data. The occupying area detection unit 214 may identifythe occupying area of the one or more objects in a check target area(for example a boundary of the workpiece 90) that moves together withthe workpiece 90. The check target area may include a plurality ofsections, and the occupying area detection unit 214 may identify anoccupying area of the one or more objects in a unit of the plurality ofsections. For example, the occupying area detection unit 214 may definethe occupying area based on a second set of the plurality of sections.

FIG. 4 illustrates the result of identifying the occupying area of therobot 3A by the occupying area detection unit 214. In the example ofFIG. 4, a check target area 93 is defined to include all the operationtarget parts 92 of the workpiece 90. The check target area 93 includes aplurality of the operation target parts 92 as a plurality of sections.The plurality of sections may not correspond to the plurality ofoperation target parts 92, and may be subdivided more than the pluralityof operation target parts 92.

The occupying area detection unit 214 identifies a containing area A01so as to include the robot body 4 of the robot 3A based on theenvironment image data, and identifies an overlapping area of the checktarget area 93 and the containing area A01 as an occupying area A02 ofthe robot 3A in the check target area 93.

At this time, the occupying area detection unit 214 identifies theoccupying area A02 in a unit of a plurality of sections. For example,the occupying area detection unit 214 identifies operation target parts92D, 92E, 92F, 92G, 92N, and 92P (a first set of the plurality ofsections) overlapping the containing area A01 among the plurality ofoperation target parts 92 as the occupying area A02 of the robot 3A inthe check target area 93.

FIG. 5 illustrates an example result of identifying an occupying area (ahuman occupying area) occupied by a worker 9 by the occupying areadetection unit 214. The occupying area detection unit 214 identifies acontaining area A11 so as to include the worker 9 based on theenvironment image data, and identifies the overlapping region of thecheck target area 93 and the containing area A11 as an occupying areaA12 of the worker 9 in the check target area 93.

At this time, the occupying area detection unit 214 identifies theoccupying area A12 in a unit of a plurality of sections. For example,the occupying area detection unit 214 identifies operation target parts92A, 92B, 92C, 92D, 92L, and 92M (a first set of the plurality ofsections) overlapping the containing area A11 among the plurality ofoperation target parts 92 as the occupying area A12 of the worker 9 inthe check target area 93.

The environment identification device 200 may be configured to outputthe occupying area of the one or more objects based on new distanceimage data, luminance image data, and an image recognition modelgenerated through machine learning so as to output the occupying area ofthe one or more objects based on input data including the distance imagedata and the luminance image data.

For example, the environment identification device 200 further includesa model storage unit 215. The model storage unit 215 stores the imagerecognition model. Specific examples of the image recognition modelinclude a neural network that outputs coordinate point sequence data (anoutput vector) representing an occupying area of one or more objects inresponse to input of distance image data and luminance image data (aninput vector).

The neural network has an input layer, one or more intermediate layers,and an output layer. The input layer outputs the input vector as it isto the next intermediate layer. The intermediate layer converts an inputfrom the previous layer with an activation function and outputs theresult to the next layer. The output layer converts an input from anintermediate layer farthest from the input layer with an activationfunction and outputs the conversion result as the output vector. Such aneural network is generated by optimizing an activation function of eachlayer based on learning data in which a teacher data set in which aninput vector and an output vector are associated with each other isaccumulated.

The occupying area detection unit 214 identifies a new occupying area ofthe one or more objects by inputting new distance image data andluminance image data to the image recognition model stored in the modelstorage unit 215.

The occupying area of the one or more objects may be identified withoutusing the image recognition model by machine learning. The occupyingarea detection unit 214 may identify the occupying area of the one ormore objects by a method that does not depend on machine learning, suchas boundary recognition in an image by pattern matching or the like.

The environment identification device 200 may be configured to storeprogress information of operation performed on the workpiece 90 and toupdate the progress information in accordance with the progress ofactual operation. For example, the environment identification device 200further includes a progress storage unit 216 and a progress managementunit 217.

The progress storage unit 216 stores progress information of theoperation performed on the operation target part 92. For example, asillustrated in FIG. 6, the progress storage unit 216 stores progressinformation of operation for each of the plurality of operation targetparts 92. The progress information in the FIG. 6 is represented by “notfinished”, “finished”, and “reserved”. “Not finished” means that theoperation has not been completed. “Finished” means that the operationhas been completed. “Reserved” means that any one of the robots 3A, 3B,and 3C is scheduled to perform operation.

The progress management unit 217 updates the progress information storedin the progress storage unit 216 in accordance with the progress of theactual operation. For example, the progress management unit 217 updatesthe progress information based on the control status of the robots 3A,3B, and 3C by robot controllers 300A, 300B, and 300C, which will bedescribed later, and the environment image data.

The robot controller 300 (a controller) controls the robots 3A, 3B, and3C to execute an operation on the workpiece 90. When the robot system 1includes a plurality of robots 3A, 3B, and 3C, the control system 100may include a plurality of robot controllers 300 that control theplurality of robots 3A, 3B, and 3C, respectively. As an example, therobot controller 300 includes the robot controllers 300A, 300B, and 300Cthat control the robots 3A, 3B, and 3C, respectively.

The robot controller 300A is configured to: identify an interlock areathat moves together with the workpiece 90 based on the current positionof the workpiece 90; determine whether the robot 3A can executeoperation on the workpiece 90 based on whether the interlock areaoverlaps an occupying area of an object other than the robot 3A; andcontrol the robot 3A to execute operation on the workpiece 90 based onthe current position of the workpiece 90 when it is determined that theoperation is possible.

As illustrated in FIG. 3, the robot controller 300A includes a programstorage unit 311, an operation target selection unit 312, a programmodification unit 313, a modified program storage unit 314, an interlockarea identification unit 315, a workability determination unit 316, anda control execution unit 317 as functional blocks.

The program storage unit 311 stores an operation program generated inadvance to control the robot 3A to execute operation on the workpiece90. For example, the program storage unit 311 stores two or moreoperation programs generated for every two or more operation targetparts 92 (hereinafter referred to as “two or more territory parts”)among the plurality of operation target parts 92. All the operationtarget parts 92 may be the territory part. The operation program mayinclude a plurality of sequential target positions of the robot 3A. Forexample, the operation program includes a plurality of motion commandsarranged in time series. Each of the plurality of motion commandsincludes a target position and a target posture of the tip 18.

The operation target selection unit 312 selects one of the two or moreterritory parts based on the progress information stored in the progressstorage unit 216, the current position of the workpiece 90 identified bythe workpiece position detection unit 213, and the current position ofthe robot 3A. For example, the operation target selection unit 312selects a part whose operation has not been completed and which islocated closest to the robot 3A from among two or more territory parts.Hereinafter, the selected part is referred to as a “selected part”.

The program modification unit 313 modifies the operation program for theselected part based on the current location of the workpiece 90. Forexample, the operation program is generated on the assumption that theworkpiece 90 is at a predetermined reference position (fixed referenceposition relative to the conveying device 2).

For example, each of the target positions of the operation program maybe associated with the fixed reference position. The programmodification unit 313 modifies the target position and the targetposture in each of the plurality of motion commands based on thedifference between the reference position and the current position.

The modified program storage unit 314 stores the operation programmodified by the program modification unit 313. Hereinafter, theoperation program stored in the modified program storage unit 314 isreferred to as a “modified program”.

The interlock area identification unit 315 identifies an interlock areathat moves together with the workpiece 90 based on the current positionof the workpiece 90. The interlock area is an area in which the presenceof another object is not allowed when the robot 3A performs theoperation on the selected part. As an example of the fact that thepresence of another object cannot be allowed, when another object existsin the interlock area, a collision between the other object and therobot 3A may occur.

The interlock area identification unit 315 may identify a plannedoccupying area of the robot 3A (an operation area occupied by the robot3A) during execution of an operation on the workpiece 90 based on themodified program, and may identify the interlock area based on theplanned occupying area. The interlock area identification unit 315 mayidentify the interlock area in a unit of a plurality of sections of thecheck target area. For example, the interlock area identification unit315 may define the interlock area based on a second set of the pluralityof sections.

FIG. 7A illustrates an example identification result of the interlockarea by the interlock area identification unit 315. Based on themodified program, the interlock area identification unit 315 identifiesa planned occupying area A21 of the robot 3A in the case of executing anoperation on an operation target part 92M that is the selected part, andidentifies the overlapping region of the check target area 93 and theplanned occupying area A21 as an interlock area A22.

At this time, the interlock area identification unit 315 identifies theinterlock area A22 in a unit of a plurality of sections. For example,the interlock area identification unit 315 identifies operation targetparts 92C, 92D, 92K, 92L, 92M, and 92N (second set of the plurality ofsections) overlapping the planned occupying area A21 among the pluralityof operation target parts 92 as the interlock area A22.

Even if the parts to be executed operations on is the same, theinterlock area may change depending on the arrangement of the robot 3A.For example, FIG. 7B illustrates that the interlock area A22 changesdepending on the arrangement of the robot 3A even when the operation isperformed on the same work target part 92M as FIG. 7A. For example, inFIG. 7B, 92D, 92E, 92L, 92M, 92N, and 92P (second set of the pluralityof sections) are selected as the interlock area A22.

The robot 3A may be configured to act on a work position. The interlockarea identification unit 315 may detect the work position based on thecurrent position of the workpiece 90 and identify the interlock area soas to surround the work position with a margin. For example, when it isallowed to set a margin large enough to include both the interlock areaA22 in FIG. 7A and the interlock area A22 in FIG. 7B, the interlock areaidentification unit 315 may set the interlock area so as to surround theoperation target position (for example, the operation target part 92M)with a predetermined margin.

For example, the interlock area surrounding the operation target part92M with a margin of two sections is identified to include operationtarget parts 92B, 92C, 92D, 92E, 92F, 92K, 92L, 92M, 92N, and 92P. Theinterlock area includes both the interlock area A22 in FIG. 7A and theinterlock area A22 in FIG. 7B.

The workability determination unit 316 may check an overlap between theinterlock area and the occupying area. For example, the workabilitydetermination unit 316 determines whether the robot 3A can perform theoperation on the workpiece 90 based on whether the interlock areaoverlaps an occupying area of an object other than the robot 3A (forexample, the robots 3B, 3C, or the worker 9). For example, when theinterlock area does not overlap the occupying area of an object otherthan the robot 3A, the workability determination unit 316 determinesthat the robot 3A can perform the operation on the selected part. On theother hand, when the interlock area overlaps the occupying area of anobject other than the robot 3A, the interlock unit determines that therobot 3A cannot perform the operation on the selected part. Theworkability determination unit 316 may check the overlap based on acomparison between the first set and the second set of the plurality ofsections.

As an example, FIG. 8 illustrates an example case where an interlockarea A31 for the operation target part 92M, which is a selected part,overlaps an occupying area A32 of the robot 3B. In this example, theworkability determination unit 316 determines that the robot 3A cannotperform the operation on the operation target part 92M.

FIG. 9 illustrates an example case where an interlock area A41 for theoperation target part 92M, which is a selected part, overlaps anoccupying area A42 of the worker 9. Also in this example, theworkability determination unit 316 determines that the robot 3A cannotexecute the operation on the operation target part 92M.

FIG. 10 illustrates an example case where an interlock area A51 for theoperation target part 92K, which is a selected part, does not overlap anoccupying area A52 of the robot 3C. In this example, the workabilitydetermination unit 316 determines that the robot 3A can execute theoperation on the operation target part 92K. As a result, the robot 3Aand the 3C can simultaneously execute operations on the same theworkpiece 90 (see FIG. 11).

The control execution unit 317 controls the robot 3A based on thecurrent position of the workpiece 90 when the workability determinationunit 316 determines that the operation is possible. For example, thecontrol execution unit 317 controls the robot 3A based on the modifiedprogram which is modified based on the current position of the workpiece90. For example, the angle target values of the joints 31, 32, 33, 34,35, and 36 are calculated by inverse kinematics calculation so as todisplace the tip 18 of the robot 3A in accordance with the modifiedprogram, and each angle of the joints 31, 32, 33, 34, 35, and 36 is madeto follow the angle target values by the actuators 41, 42, 43, 44, 45,and 46.

The robot controller 300 may be configured to control the robot 3Afurther based on operation target image data generated by the operationtarget camera 51 provided in the robot 3A. For example, the robotcontroller 300 may further include an operation object positiondetection unit 318.

The operation object position detection unit 318 acquires operationtarget image data generated by the operation target camera 51. Thecontrol execution unit 317 controls the robot 3A further based on theoperation target image data acquired by the operation object positiondetection unit 318. For example, the control execution unit 317identifies a positional deviation (for example, a deviation of theposition and posture of the tip 18 with respect to the selected part)based on the operation target image data, corrects the target positionand target posture of the tip 18 so as to reduce (compensate) theidentified deviation, and controls the robot 3A based on the correctedtarget position and target posture.

Although the example configuration in which the workabilitydetermination unit 316 determines that the operation is possible andthen the robot 3A starts the operation on the selected part has beendescribed above, the configuration is not limited thereto. The controlexecution unit 317 may pause the robot 3A after controlling the robot 3Ato execute the operation on the selected part halfway, and wait for theworkability determination unit 316 to determine that the operation ispossible.

The control execution unit 317 may generate an operation programcorresponding to a current position of the workpiece 90 in real timewithout being based on an operation program generated in advance, andcontrol the robot 3A based on the generated operation program. In thisexample, the program modification unit 313 and the modified programstorage unit 314 can be omitted.

Similarly to the robot controller 300A, the robot controller 300B isconfigured to: identify an interlock area that moves together with theworkpiece 90 based on the current position of the workpiece 90;determine whether the robot 3B can execute operation on the workpiece 90based on whether the interlock area overlaps an occupying area of anobject other than the robot 3B; and control the robot 3B to executeoperation on the workpiece 90 based on the current position of theworkpiece 90 when it is determined that the operation is possible.

A robot controller 300C is configured to: identify an interlock areathat moves together with the workpiece 90 based on the current positionof the workpiece 90; determine whether the robot 3C can executeoperation on the workpiece 90 based on whether the interlock areaoverlaps an occupying area of an object other than the robot 3C; andcontrol the robot 3C to execute operation on the workpiece 90 based onthe current position of the workpiece 90 when it is determined that theoperation is possible.

The configurations of the robot controller 300B and the 300C are thesame as those of the robot controller 300A. Therefore, detaileddescription of the robot controller 300B and the 300C is omitted.

FIG. 12 is a block diagram illustrating a hardware configuration of theenvironment identification device 200 and the robot controller 300. Asillustrated in FIG. 12, the environment identification device 200includes circuitry 290. The circuitry 290 includes one or moreprocessor(s) 291, a memory 292, storage 293, a communication port 294,and an input/output port 295. The storage 293 includes acomputer-readable storage medium such as a nonvolatile semiconductormemory. The storage 293 stores a program for controlling the environmentidentification device 200 to identify a current position of theworkpiece 90 in the conveying device 2 and an occupying area of one ormore objects in the vicinity of the workpiece 90. For example, thestorage 293 stores a program for configuring the above-describedfunctional blocks in the environment identification device 200.

The memory 292 temporarily stores the program loaded from the storagemedium of the storage 293 and the calculation result by the processor291. The processor 291 configures each functional block of theenvironment identification device 200 by executing the program incooperation with the memory 292. The communication port 294 communicateswith the robot controller 300 via a network NW in accordance withinstructions from the processor 291. The input/output port 295 inputsand outputs information to and from the luminance camera 7 and thedistance camera 8 in accordance with instructions from the processor291.

The robot controller 300 includes circuitry 390. The circuitry 390includes one or more processor(s) 391, a memory 392, storage 393, acommunication port 394, and a driver circuit 395. The storage 393includes a computer-readable storage medium such as a nonvolatilesemiconductor memory. The storage 393 stores a program for controllingthe robot controller 300 to: identify an interlock area that movestogether with the workpiece 90 based on the current position of theworkpiece 90; determine whether the robots 3A, 3B, and 3C can executeoperation on the workpiece 90 based on whether the interlock areaoverlaps an occupying area of an object other than the robots 3A, 3B,and 3C; and control the robots 3A, 3B, and 3C to execute operation onthe workpiece 90 based on the current position of the workpiece 90 whenit is determined that the operation is possible. For example, thestorage 393 stores a program for configuring the above-describedfunctional blocks in the robot controller 300.

The memory 392 temporarily stores the program loaded from the storagemedium of the storage 393 and the calculation result by the processor391. The processor 391 configures each functional block of the robotcontroller 300 by executing the program in cooperation with the memory392. The communication port 394 communicates with the environmentidentification device 200 via the network NW in accordance withinstructions from the processor 391. The driver circuit 395 outputsdrive power to the robot 3A, 3B, and 3C in accordance with instructionsfrom the processor 391.

It should be noted that the circuitry 290 and 390 may not be limited toone that configures each function by a program. For example, thecircuitry 290 and 390 may configure at least a part of the functions bydedicated logic circuitry or an application specific integrated circuit(ASIC) in which the dedicated logic circuitry is integrated.

Control Procedure

Next, an example control procedure executed by the control system 100will be described as an example of the control method. This procedureincludes an environment identification procedure executed by theenvironment identification device 200 and a robot control procedureexecuted by the robot controller 300. Hereinafter, each procedure willbe described in detail.

Environment Identification Procedure

As illustrated in FIG. 13, the environment identification device 200executes operations S01, S02, and S03. In the operation S01, the imagedata acquisition unit 211 acquires the environment image data generatedby the environment imaging device 6. In the operation S02, the workpieceposition detection unit 213 identifies the current position of theworkpiece 90 based on the environment image data. In the operation S03,the occupying area detection unit 214 identifies an occupying area ofone or more objects based on the environment image data. The environmentidentification device 200 repeats the above procedure.

Robot Control Procedure

The robot control procedure is common to the robot controllers 300A,300B, and 300C. Accordingly, the robot control procedure executed by therobot controller 300A will be described below, and the description ofthe robot control procedures executed by the robot controller 300B and300C will be omitted.

This procedure includes: identifying an interlock area that movestogether with the workpiece 90 based on the current position of theworkpiece 90; determining whether the robot 3A can execute operation onthe workpiece 90 based on whether the interlock area overlaps anoccupying area of an object other than the robot 3A; and controlling therobot 3A to execute the operation on the workpiece 90 based on thecurrent position of the workpiece 90 when it is determined that theoperation is possible.

As illustrated in FIG. 14, the robot controller 300 first executesoperations S11, S12, S13, S14, and S15. In the operation S11, theoperation target selection unit 312 selects one of two or more territoryparts based on the progress information stored in the progress storageunit 216, the current position of the workpiece 90 identified by theworkpiece position detection unit 213, and the current position of therobot 3A. In the operation S12, the operation target selection unit 312requests the progress management unit 217 to change the selected part to“reserved”. The progress management unit 217 updates the progressinformation of the progress storage unit 216 so as to set the selectedpart to “reserved”. In the operation S13, the program modification unit313 modifies the operation program for the selected part based on thecurrent position of the workpiece 90.

In the operation S14, the interlock area identification unit 315identifies an interlock area that moves together with the workpiece 90based on the current position of the workpiece 90. In the operation S15,based on whether the interlock area overlaps the occupying area ofanother object other than the robot 3A (for example, the robot 3B, 3C,or the worker 9), the workability determination unit 316 determineswhether the robot 3A can execute the operation on the workpiece 90.

If it is determined in the operation S15 that the robot 3A cannotexecute the operation on the workpiece 90, the robot controller 300returns the processing to the operation S13. Thereafter, themodification of the operation program based on the current position ofthe workpiece 90, the identification of the interlock area based on thecurrent position of the workpiece 90, and the determination of whetherthe operation is possible are repeated until it is determined that therobot 3A can execute the operation on the workpiece 90.

If it is determined in the operation S15 that the robot 3A can executethe operation on the workpiece 90, the robot controller 300 repeatedlyexecutes control of the robot 3A based on the modified program in apredetermined control cycle.

First, the robot controller 300 executes operations S21 and S22. In theoperation S21, the control execution unit 317 calculates the targetposition and target posture of the tip 18 in the current control cyclebased on the modified program. In the operation S22, the controlexecution unit 317 checks whether the tip 18 has reached a predeterminedfine adjustment start position. As an example, the fine adjustment startposition is a position at which at least a part of the selected part isincluded in the operation target image data.

If it is determined in the operation S22 that the tip 18 has not reachedthe fine adjustment start position, the robot controller 300 executesoperations S25 and S26. In the operation S25, the actuators 41, 42, 43,44, 45, and 46 drives the joints 31, 32, 33, 34, 35, and 36 so that thecontrol execution unit 317 controls the tip 18 to follow the calculatedtarget position and target posture. In the operation S26, the controlexecution unit 317 checks whether control according to all motioncommands of the operation program is completed.

If it is determined in the operation S26 that the control according to apart of the motion commands of the operation program is not completed,the robot controller 300 returns the processing to the operation S21.Thereafter, the control of the robot 3A based on the modified program isrepeated in the above-described control cycle until the controlaccording to all the motion commands is completed.

In this repetition, if it is determined in the operation S22 that thetip 18 has reached the fine adjustment start position, the robotcontroller 300 executes operations S23 and S24 before the operation S25.In the operation S23, the operation object position detection unit 318acquires operation target image data generated by the operation targetcamera 51. In the operation S24, the control execution unit 317identifies the deviation of the position and posture of the tip 18 withrespect to the selected part based on the operation target image data,and corrects the target position and target posture of the tip 18 so asto reduce the identified deviation. Thereafter, the control of the robot3A based on the modified program and the operation target image data isrepeated in the above-described control cycle until the controlaccording to all the motion commands is completed.

If is it determined in the operation S26 that control according to allmotion commands of the operation program is completed, the robotcontroller 300 executes a operation S27. In the operation S27, theoperation target selection unit 312 requests the progress managementunit 217 to change the selected part to “finished”. The progressmanagement unit 217 updates the progress information of the workabilitydetermination unit 316 so as to set the selected part to “finished”.Thus, the robot control procedure is completed.

Effects of Present Embodiment

As described above, the robot system 1 includes the conveying device 2configured to convey the workpiece 90; the environment identificationdevice 200 configured to identify the current position of the workpiece90 and the occupying area of one or more objects; and the robotcontroller 300 configured to control the robots 3A, 3B, and 3C toexecute the operation on the workpiece 90, and the robot controller 300includes: the interlock area identification unit 315 configured toidentify, based on the current position of the workpiece 90, theinterlock area moving together with the workpiece 90; the workabilitydetermination unit 316 configured to determine whether the robot 3A, 3B,and 3C can execute the operation on the workpiece 90 based on whetherthe interlock area overlaps the occupying area of an object other thanthe robots 3A, 3B, and 3C; and the control execution unit 317 configuredto control the robots 3A, 3B, and 3C based on the current position ofthe workpiece 90 when the workability determination unit 316 determinesthat the operation is possible.

The robot system 1 determines whether the robots 3A, 3B, and 3C canexecute the operation on the workpiece 90 based on whether the interlockarea that moves together with the workpiece 90 interferes with anoccupying area of another object. Therefore, even in a situation wherethe workpiece 90 as the operation target is transported by the conveyingdevice 2 and moves, when the robot execute the operation on theworkpiece 90 as the operation target, whether the robots 3A, 3B, and 3Cmay collide with another object may be evaluated and whether the robots3A, 3B, and 3C can execute the operation may be determined. Therefore,the robot system 1 may flexibly adapt to environmental changes.

The robot controller 300 may further include the program modificationunit 313 configured to modify the predetermined operation program basedon the current position of the workpiece 90, and the control executionunit 317 may be configured to control the robots 3A, 3B, and 3C based onthe modified operation program when the workability determination unit316 determines that the operation is possible. The position of theworkpiece 90 when it is determined that the operation is possiblechanges depending on the timing at which the operation becomes possibleduring conveyance. Accordingly, the motions of the robots 3A, 3B, and 3Cshould be adapt to the position of the workpiece 90 when it isdetermined that the operation is possible. On the other hand, the robotsystem 1 modifies a predetermined operation program based on the currentposition of the workpiece 90. Therefore, the operations of the robots3A, 3B, and 3C can be adapted to the position of the workpiece 90 whenit is determined that the operation is possible.

The environment identification device 200 may be configured to identifyan occupying area of one or more objects in the check target area 93that moves together with the workpiece 90. In this example, theprocessing load in the robot controller 300 may be reduced by reducingthe area to be checked for overlap with the interlock area among the oneor more occupying areas.

The robot system 1 may further include the robot controller 300 (asecond controller) for controlling the robot 3A, 3B, and 3C (a secondrobot) to execute the operation on the workpiece 90 in an operationrange overlapping the operation range of the robots 3A, 3B, and 3C, andthe second controller may include: the interlock area identificationunit 315 configured to identify a second interlock area that movestogether with the workpiece 90 based on the current position of theworkpiece 90; a workability determination unit (a second workabilitydetermination unit) configured to determine whether the second robot canexecute the operation on the workpiece 90 based on whether the secondinterlock area overlaps an occupying area of an object other than thesecond robot; and the control execution unit 317 configured to controlthe second robot based on the current position of the workpiece 90 whenthe second workability determination unit determines that operation ispossible. In this example, the flexible adaptability to environmentalchanges may be utilized for adjustment of operation timings of the robot3A, 3B, and 3C whose operation ranges overlap each other.

The occupying area of the object other than the robot 3A, 3B, and 3C mayinclude the human occupying area occupied by the worker 9. In thisexample, whether operations by the robots 3A, 3B, and 3C is possible maybe determined even in an operation environment where a person cooperateswith the robots 3A, 3B, and 3C.

The robot system 1 may further include the environment imaging device 6configured to image the target area including the workpiece conveyed bythe conveying device 2 to generate the environment image data, and theenvironment identification device 200 may be configured to identify thecurrent location of the workpiece 90 and the occupying area of one ormore objects based on the environment image data. In this example, byimproving the reliability of the environment information, thereliability of the operations of the robots 3A, 3B, and 3C based on theenvironment information may be improved.

The environment imaging device 6 may be configured to generate theenvironment image data including the distance image data and theluminance image data for the target area, and the environmentidentification device 200 may be configured to identify the occupyingarea of the one or more objects based on the distance image data and theluminance image data. In this example, the reliability of theenvironment information can be further improved by using both thedistance image data and the luminance image data.

The environment identification device 200 may be configured to recognizethe marker 91 attached to the workpiece 90 based on the environmentimage data, and to identify the current position of the workpiece 90based on the position of the marker 91 in the target area. In thisexample, the environment image data can also be used to detect theposition of the workpiece 90.

The interlock area identification unit 315 may be configured to identifythe planned occupying area of the robots 3A, 3B, and 3C in the case ofexecuting the operation on the workpiece 90 based on the modifiedoperation program, and to identify the interlock area based on theplanned occupying area. In this example, a case where it is determinedthat the operation is impossible although the operation is actuallypossible may be reduced, and the operation efficiency of the robots 3A,3B, and 3C may be improved.

The interlock area identification unit 315 may be configured to identifythe interlock area so as to surround the operation target positions ofthe robots 3A, 3B, and 3C with a predetermined margin based on thecurrent position of the workpiece 90. In this example, the process ofidentifying the interlock area can be simplified, and the processingload on the robot controller 300 can be reduced.

The check target area may include a plurality of sections, theenvironment identification device 200 may be configured to the occupyingarea of the one or more objects in units of the plurality of sections,and the interlock area identification unit 315 may be configured toidentify the interlock area in units of the plurality of sections. Inthis example, the process of identifying one or more robot overlappingareas and the process of identifying the interlock area may besimplified, and the processing load in the environment identificationdevice 200 and the robot controller 300 may further be reduced.

The environment identification device 200 may be configured to identifya new occupying area of the one or more objects based on new distanceimage data, new luminance image data, and an image recognition modelgenerated through machine learning so as to output the occupying area ofthe one or more objects based on input data including the distance imagedata and the luminance image data. In this example, one or more robotoccupying areas may be reliably identified based on the distance imagedata and the luminance image data by utilizing the machine learning.

The robots 3A, 3B, and 3C may include the robot body 4, the conveyingcarriage 5 that holds and conveys the robot body 4, and the operationtarget camera 51 (the operation target imaging device) configured toimage the operation area of the robot body to generate the operationtarget image data, and the control execution unit 317 may be configuredto control the robots 3A, 3B, and 3C further based on the operationtarget image data. In this example, the operation accuracy of the robots3A, 3B, and 3C with respect to the workpiece 90 may be maintained evenin a situation where a positional deviation of the conveying carriage 5may occur.

It is to be understood that not all aspects, advantages and featuresdescribed herein may necessarily be achieved by, or included in, any oneparticular example. Indeed, having described and illustrated variousexamples herein, it should be apparent that other examples may bemodified in arrangement and detail.

What is claimed is:
 1. A robot system comprising: a conveying deviceconfigured to convey a workpiece; a robot configured to execute anoperation on the workpiece; and circuitry configured to: identify acurrent position of the workpiece and an object area occupied by anobject; identify an interlock area that moves with the current positionof the workpiece being conveyed by the conveying device; check anoverlap between the interlock area and the object area; and control therobot to execute the operation based on the current position of theworkpiece in response to determining that the interlock area does notoverlap the object area.
 2. The robot system according to claim 1,wherein the circuitry is configured to: store a program for controllingthe robot to execute the operation on the workpiece; modify the programbased on the current position of the workpiece for controlling the robotto execute the operation on the workpiece being conveyed by theconveying device; and control the robot based on the modified program inresponse to determining that the interlock area does not overlap theobject area.
 3. The robot system according to claim 2, wherein thecircuitry is configured to: identify, based on the modified program, anoperation area occupied by the robot during the operation on theworkpiece; and identify the interlock area based on the operation area.4. The robot system according to claim 3, wherein the workpiece includesa boundary, and wherein the circuitry is configured to identify theinterlock area based on a part of the operation area positioned withinthe boundary.
 5. The robot system according to claim 2, wherein theprogram comprises a plurality of sequential target positions of therobot each of which is associated with a fixed reference positionrelative to the conveying device, and wherein the modified operationprogram includes a modification of at least one of the target positionsbased on a difference between the current position of the workpiecebeing conveyed by the conveying device and the fixed reference position.6. The robot system according to claim 1, wherein the workpiece includesa boundary, and wherein the circuitry is configured to identify theobject area within the boundary of the workpiece.
 7. The robot systemaccording to claim 6, wherein the circuitry is configured to check theoverlap between the interlock area and the object area within theboundary of the workpiece.
 8. The robot system according to claim 6,wherein the boundary includes a plurality of sections, wherein thecircuitry is configured to: define the object area based on a first setof the plurality of sections; define the interlock area based on asecond set of the plurality of sections; and check the overlap based ona comparison between the first set and the second set.
 9. The robotsystem according to claim 1, wherein the object is a second robotconfigured to execute a second operation on the workpiece.
 10. The robotsystem according to claim 1, wherein the object area is a human areaoccupied by a human worker.
 11. The robot system according to claim 1,wherein the circuitry is configured to identify the current position ofthe workpiece and the object area based on image data of imaging targetincluding the workpiece conveyed by the conveying device and the object,and wherein the image data comprises: distance image data includingfirst pixels each of which is associated with a distance from a camerato a portion of the imaging target; and luminance image data includingsecond pixels each of which is associated with a luminance of a portionof the imaging target.
 12. The robot system according to claim 11,wherein the circuitry is configured to identify the object area based onthe distance image data, the luminance image data, and an imagerecognition model generated through machine learning so as to output theobject area in response to an input of the distance image data and theluminance image data.
 13. The robot system according to claim 1, whereinthe circuitry is configured to: recognize a marker attached to theworkpiece based on image data of the workpiece; and identify the currentposition of the workpiece based on a position of the marker in the imagedata.
 14. The robot system according to claim 1, wherein the robot isconfigured to act on a work position of the workpiece during theoperation, and wherein the circuitry is further configured to: identifythe work position based on the current position of the workpiece beingconveyed by the conveying device; and identify the interlock area so asto surround the work position with a margin.
 15. The robot systemaccording to claim 1, further comprising an imaging device fixed to therobot configured to image the workpiece, and wherein the circuitry isconfigured to: receive an image data of the workpiece from the imagingdevice; and modify the operation based on the image data of theworkpiece.
 16. The robot system according to claim 15, furthercomprising a mobile carriage that holds and conveys the robot, whereinthe circuitry is configured to modify the operation to compensate, basedon the image data of the workpiece, a positional deviation of the robotwith respect to the workpiece.
 17. A controller comprising a circuitryconfigured to: identify an interlock area that moves with a currentposition of a workpiece being conveyed by a conveying device; check anoverlap between the interlock area and an object area occupied by anobject; and control a robot to execute an operation on the workpiecebased on the current position of the workpiece in response todetermining that the interlock area does not overlap the object area.18. A control method comprising: identifying an interlock area thatmoves with a current position of a workpiece being conveyed by aconveying device; checking an overlap between the interlock area and anobject area occupied by an object; and controlling a robot to execute anoperation on the workpiece based on the current position of theworkpiece in response to determining that the interlock area does notoverlap the object area.
 19. The control method according to claim 18,wherein said checking includes checking the overlap between theinterlock area and the object area within a boundary of the workpiece.20. The control method according to claim 18, further comprising:storing a program for controlling the robot to execute the operation onthe workpiece; modifying the program based on the current position ofthe workpiece for controlling the robot to execute the operation on theworkpiece being conveyed by the conveying device, wherein saidcontrolling includes controlling the robot based on the modifiedoperation program in response to determining that the interlock areadoes not overlap the object area.